EE401 (Semester 1, 2011) Jitkomut Songsiri 

1. Review of Probability 



Random Experiments 



The Axioms of Probability 
Conditional Probabilty 



Independence of Events 



Sequential Experiments 



i-i 



Random Experiments 

An experiment in which the outcome varies in an unpredictable fashion 
when the experiment is repeated under the same conditions 

Examples: 

• Select a ball from an urn containing balls numbered 1 ton 

• Toss a coin and note the outcome 

• Roll a dice and note the outcome 

• Measure the time between page requests in a Web server 

• Pick a number at random between and 1 
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Sample space 

Sample space is the set of all possible outcomes, denoted by S 



obtained by listing all the elements, e.g., S = {H,T}, or 

giving a property that specifies the elements, e.g., S = {x | < x < 3} 



Same experimental procedure may have different sample spaces 
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Experiment 1: Pick two numbers at random between zero and one 

Experiment 2: Pick a number X at random between and 1, then pick 
a number Y at random between and X 
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Three possibilities for the number of outcomes in sample spaces 

finite, countably infinite, uncountably infinite 
Examples: 

5i = {l,2,3,...,10} 

5 2 = {HH,HT,TT,TH} 

5 3 = { x £ Z | < x < 10} 

54 = {1, 2, 3, . . .} 

5 5 = {( x , y)eRxR\0<y<x<l} 

Sq = Set of functions X(t) for which X(t) = for t > to 

Discrete sample space: if S is countable (£1, S2, £3, £4) 
Continuous sample space: if S is not countable (S^Sq) 
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Events 



Event is a subset of a sample space when the outcome satisfies certain 
conditions 



Examples: A^ denotes an event corresponding to the experiment Ek 

Ei : Select a ball from an urn containing balls numbered 1 to 10 
A\ : An even-numbered ball (from 1 to 10) is selected 

5x = {1,2, 3,..., 10}, Ax = {2, 4, 6, 8, 10} 

E2 : Toss a coin twice and note the sequence of heads and tails 
A 2 : The two tosses give the same outcome 

S 2 = {HH,HT,TT,TH}, a 2 = {HH,TT} 
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Es : Count # of voice packets containing only silence from 10 speakers 
A 3 : No active packets are produced 

S 3 = {xeZ\0<x< 10}, A 3 = {0} 



Two events of special interest: 

• Certain event, S, which consists of all outcomes and hence always 
occurs 

• Impossible event or null event, 0, which contains no outcomes and 
never occurs 
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Review of Set Theory 

A = B if and only \f A c B and B c A 

A U B (union): set of outcomes that are in A or in B 

An B (intersection): set of outcomes that are in A and in B 

A and B are disjoint or mutually exclusive if A D B = 

t4 c (complement): set of all elements not in A 

AuB = BuAand AnB = BnA 

AU (B U C) = (AU B) U C and An (B n C) = (An B) n C 

Au(BnC) = {AuB)n(AuC) and An(BuC) = (AnB)u(AnC) 

DeMorgan's Rules 

(A U Bf = A c n B c , (A n Bf = A C UB C 
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Axioms of Probability 



Probabilities are numbers assigned to events indicating how likely it is that 
the events will occur 



A Probability law is a rule that assigns a number P(A) to each event A 



P(A) is called the the probability of A and satisfies the following axioms 

Axiom 1 P(A) > 
Axiom 2 P(S) = 1 
Axiom 3 If A n B = then P(A U B) = P(A) + P(B) 
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Probability Facts 



P(A C ) = 1 - P{A) 
P(A) < 1 
P(0) = 



If Ai : A<2, • • • , A n are pairwise mutually exclusive then 



n \ n 



P[{JA k )=Y,P{Ak) 



k fc=l / fc = l 



P(A U B) = P(A) + P(B) - P(A n 5) 
If Ac B then P(A) < P(B) 
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Conditional Probability 



The probability of event A given that event B has occured 



The conditional probability, P(A\B), is defined as 

P(A\B) = P ^*\ forP(B)>0 



If B is known to have occured, then A can occurs only if A D B occurs 



Simply renormalizes the probability of events that occur jointly with B 



Useful in finding probabilities in sequential experiments 
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Example: Tree diagram of picking balls 

Selecting two balls at random without replacement 




W\ Outcome of first draw 



vv2 Outcome of second draw 




10 



_3_ _3_ 

10 10 



3_ 

10 



Bi,B2 are the events of getting a black ball in the first and second draw 



P(B 2 \B 1 ) = ± P(W 2 \B 1 ) = 1 P(B 2 \W 1 ) = ^ P(W 2 \W 1 ) = 1 



The probability of a path is the product of the probabilities in the transition 



P(B 1 n B 2 ) = P( J B 2 | J B 1 )P( J B 1 ) = \l= l 



45 10 
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Example: Tree diagram of Binary Communication 



Output 




(l—p)(l — e) (1 — p)e pe 



p(l-e) 



A{\ event the input was i, 



By. event the reciever was i 



P(A n B ) 
P(A n fix) 
P(A 1 n B ) 



(l-p)(l- e ) 

{l-p)e 

pe 

p(l-e) 
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Theorem on Total Probability 

Let £?i, B 2 , • • • , B n be mutually exclusive events such that 



S = J BiU5 2 U---U5 



n 



(their union equals the sample space) 
Event A can be partitioned as 

A = AnS = (AnB 1 )u(AnB 2 )U---U(AnB n ) 

Since An B^ are disjoint, the probability of A is 

p(A) = P(A n Bi) + P(A n B 2 ) + • • • + P(A n B n ) 

or equivalently, 

P(A) = P{A\B l )P{B l ) + P(A\B 2 )P(B 2 ) + • • • + P(A|£ n )P(5 n ) 
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Example: revisit the tree diagram of picking two balls 



W\ Outcome of first draw 




Vv2 Outcome of second draw 




10 



10 10 



3_ 

10 



Find the probability of the event that the second ball is white 



P(W 2 ) = P(W 2 \B 1 )P(B 1 ) + P{W 2 \W 1 )P(W 1 ) 

_ 3 2 1 3 _ 3 

~ I' 5 + 2 ' 5 ~ 5 
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Bayes' Rule 

The conditional probablity of event A given B is related to the inverse 
conditional probability of event B given A by 

• P(A) is called a priori probability 

• P(A\B) is called a posteriori probability 

Let Ai : A 2 , . . . , A n be a partition of S 

r^Al^ P(B AAPiAi) 

P(Ai\B) = 



Ek=iP(B\A k )P(A k ) 
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Example: Binary Channel 




1-e 



Output 







Ai event the input was i 
Bi event the receiver output was i 
Input is equally likely to be or 1 



1 



P(Bi) = P(B 1 \A )P(A )+P(B 1 \A 1 )P(A 1 ) = e(l/2) + (l-e)(l/2) = 1/2 



Applying Bayes' Rule, we obtain 



P(A \B 1 ) = ^m^^ = ^ = e 



P(B!) 



1/2 



If e < 1/2, input 1 is more likely than when 1 is observed 
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Independence of Events 



Events A and B are independent if 



P(ADB) = P(A)P(B) 



Knowledge of event B does not alter the probability of event A 



This implies P(A\B) = P{A) 





1 1 

A and B are independent A and C are not independent 
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Example: System Reliability 



( Unit 2 J 



( Unit 3 Y*- 



Controller 



/ Unit 1 J 



System is 'up' if the controller and at 
least two units are functioning 

Controller fails with probability p 

Peripheral unit fails with probability a 

All components fail independently 



A: event the controller is functioning 

Bi m . event unit i is functioning 

F: event two or more peripheral units are functioning 



Find the probability that the system is up 
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The event F can be partition as 

F = (B 1 nB 2 n B c 3 ) u (Si n^n B 3 ) u {B\ nB 2 n B 3 ) u (B 1 nB 2 n B 3 ) 



Thus, 



P(F) = P(B 1 )P(B 2 )P(Bl) + P(B 1 )P(B C 2 )P(B 3 ) 
+ P(B[)P(B 2 )P(B 3 ) + P{B 1 )P{B 2 )P{B 3 ) 
= 3(l-a) 2 a+(l-a) 3 



P(system is up) = P(A n F) = P(A)P(F) 

= (1 - p)P(F) = (1 - p){3(l - a) 2 a + (1 - a) 3 } 
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Sequential Independent Experiments 



Consider a random experiment consisting of n independent experiments 

Let Ai : A2, . . • , A n be events of the experiments 

We can compute the probability of events of the sequential experiment 

p(A l n A 2 n • • • n A n ) = p(A 1 )p(a 2 ) . . . P(A n ) 



Example: Bernoulli trial 

— Perform an experiment and note if the event A occurs 

— The outcome is "success" or "failure" 

— The probability of success is p and failure is 1 — p 
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Binomial Probability 



Perform n Bernoulli trials and observe the number of successes 

Let X be the number of successes in n trials 

The probability of X is given by the Binomial probability law 

for k = 0, 1, . . . , n 
The binomial coefficient 



n n\ 



k J k\(n-k)\ 
is the number of ways of picking k out of n for the successes 
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Example: Error Correction Coding 




Transmit each bit three times 

Decoder takes a majority vote of the received 
bits 



1-e 



Compute the probability that the receiver makes an incorrect decision 

• View each transmission as a Bernoulli trial 

• Let X be the number of wrong bits from the receiver 

P(X>2)=( 3 2 )eHl-e) + (l)e* 
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Mutinomial Probability 



Generalize the binomial probability law to the occurrence of more than 
one event 

Let B\ : B2, . . . , B m be possible events with 

P(B k )=p k , and pi+p 2 H hp m = l 



Suppose n independent repetitions of the experiment are performed 

Let Xj be the number of times each Bj occurs 

The probability of the vector (Xi, X2, . . . , X m ) is given by 

P{Xx = fci, X 2 = k 2 ,...,X m = k m ) = - - , ' , - VxVj ■■■vl m 



m 



where k\ + fc 2 + • • • + fc m = n 
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Geometric Probability 

Repeat independent Bernoulli trials until the the first success occurs 
Let X be the number of trials until the occurrence of the first success 
The probability of this event is called the geometric probability law 

P(X = k) = (1 - p) fc "V for Jfc = 1, 2, . , . 



The geometric probabilities sum to 1: 



oo oo 



V 



*-^ *-^ 1 — q 

fc=i fe=i H 

where q = 1 — p 

The probability that more than n trials are required before a success 

P(X>n) = (l-p) n 
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Example: Error Control by Retransmission 



• A sends a message to B over a radio link 

• B can detect if the messages have errors 

• The probability of transmission error is q 

• Find the probability that a message needs to be transmitted more than 
two times 

Each transmission is a Bernoulli trial with probability of success p = 1 — q 

The probability that more than 2 transmissions are required is 

P(X >2)=q 2 
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Sequential Dependent Experiments 

Sequence of subexperiments in which the outcome of a given 
subexperiment determine which subexperiment is performed next 

Example: Select the urn for the first draw by flipping a fair coin 










Draw a ball, note the number on the ball and replace it back in its urn 



The urn used in the next experiment depends on # of the ball selected 
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Trellis Diagram 



Sequence of outcomes 




Probability of a sequence of outcomes 




is the product of probabilities along the path 
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Markov Chains 

Let Ai, A 2 , • • • , A n be a sequence of events from n sequential experiments 
The probability of a sequence of events is given by 

P{A 1 A 2 '"A n ) = P(A n \A 1 A 2 ■ • • A n _ 1 )P{A 1 A 2 ■ ■ ■ A n _x) 

If the outcome of A n _\ only determines the n th experiment and A n then 

P(A n \A 1 A 2 • • • An-!) = P{A n \A n ^ 1 ) 

and the sequential experiments are called Markov Chains 
Thus, 

P{A 1 A 2 • • • A^ = P(A n \ A n _ 1 )P{A n _ 1 \An-2) • • • P{A 2 \A 1 )P{A 1 ) 
Find P(0011) in the urn example 
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The probability of the sequence 0011 is given by 



P(0011) = P(1|1)P(1|0)P(0|0)P(0) 



where the transition probabilities are 



p(i|i) = |, P(i\o) = \, P(o\o) = l 



and the initial probability is given by 



P(0) = \ 



Hence, 

,5121 5 
P(0011) =-•-•-•- = 



6 3 3 2 54 
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